Generating robotic grasping instructions for inventory items

ABSTRACT

Robotic arms may be utilized to grasp inventory items within an inventory system. Information about an inventory item to be grasped can be detected and used to determine a grasping strategy in conjunction with information from a database. Instructions for grasping an inventory item can be generated based on the detected information and the database.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.14/572,420, filed on Dec. 16, 2014, titled “GENERATING ROBOTIC GRASPINGINSTRUCTIONS FOR INVENTORY ITEMS”, which is related to and incorporatesby reference for all purposes the full disclosure of co-pending U.S.patent application Ser. No. 14/572,332, filed Dec. 16, 2014, entitled“ROBOTIC GRASPING OF ITEMS IN INVENTORY SYSTEM”, the disclosures ofwhich are herein incorporated in their entireties.

BACKGROUND

Modern inventory systems, such as those in mail order warehouses, supplychain distribution centers, airport luggage systems, and custom-ordermanufacturing facilities, face significant challenges in responding torequests for inventory items. As inventory systems grow, the challengesof simultaneously completing a large number of packing, storing, andother inventory-related tasks become non-trivial. In inventory systemstasked with responding to large numbers of diverse inventory requests,inefficient utilization of system resources, including space, equipment,and manpower, can result in lower throughput, unacceptably long responsetimes, an ever-increasing backlog of unfinished tasks, and, in general,poor system performance. Additionally, expanding or reducing the size orcapabilities of many inventory systems requires significant changes toexisting infrastructure and equipment. As a result, the cost ofincremental changes to capacity or functionality may be prohibitivelyexpensive, limiting the ability of the system to accommodatefluctuations in system throughput.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments in accordance with the present disclosure will bedescribed with reference to the drawings, in which:

FIG. 1 illustrates an example of an inventory system having a roboticarm configured for grasping inventory items of the inventory systemaccording to a particular embodiment;

FIG. 2 illustrates components of an inventory system according toparticular embodiments;

FIG. 3 illustrates in greater detail the components of an examplemanagement module that may be utilized in particular embodiments of theinventory system shown in FIG. 2;

FIGS. 4 and 5 illustrate in greater detail an example mobile drive unitthat may be utilized in particular embodiments of the inventory systemshown in FIG. 2;

FIG. 6 illustrates in greater detail an example inventory holder thatmay be utilized in particular embodiments of the inventory system shownin FIG. 2;

FIG. 7 illustrates in greater detail additional components of an examplemanagement module of FIG. 3 according to a particular embodiment;

FIG. 8 illustrates an example of a user interface for receiving humaninput for a grasping strategy according to particular embodiments;

FIG. 9 illustrates an example of a station utilizing a robotic arm forhandling of inventory items according to particular embodiments;

FIG. 10 illustrates in greater detail an example of components that maybe utilized in directing the robotic arm of FIG. 9 to an item releaselocation according to particular embodiments;

FIG. 11 illustrates a cross-sectional view of a tray of the station ofFIG. 9 according to particular embodiments;

FIG. 12 illustrates examples of inventory system components with which arobotic arm may interact according to particular embodiments;

FIG. 13 illustrates an environment in which various features of theinventory system can be implemented, in accordance with at least oneembodiment.

DETAILED DESCRIPTION

In the following description, various embodiments will be described. Forpurposes of explanation, specific configurations and details are setforth in order to provide a thorough understanding of the embodiments.However, it will also be apparent to one skilled in the art that theembodiments may be practiced without the specific details. Furthermore,well-known features may be omitted or simplified in order not to obscurethe embodiment being described.

Embodiments herein are directed to an inventory system having multipleinventory holders and drive units for moving the inventory holders.Specifically, features herein are directed to robotic arms ormanipulators and associated components that may facilitate the movementof inventory items and other features of the inventory system among andbetween elements of the inventory system. The robotic arms ormanipulators may be controlled so as to grasp items in a mannerparticularly suited for grasping of a target item. For example, a targetitem, or characteristics thereof, may be identified, such as by opticalor other sensors, in order to determine a grasping strategy for theitem. The grasping strategy may be based at least in part upon adatabase containing information about the item, characteristics of theitem, and/or similar items, such as information indicating graspingstrategies that have been successful or unsuccessful for such items inthe past. Entries or information in the database may be originatedand/or updated based on human input for grasping strategies, determinedcharacteristics of a particular item, and/or machine learning related tograsping attempts of other items sharing characteristics with theparticular item. Embodiments herein include aspects directed togenerating and/or accessing such databases.

Thus, in an illustrative example according to one embodiment, a roboticarm is positioned within reach of a tray. A human operator loads thetray with an inventory item (for example a coffee mug) that is to begrasped by the robotic arm and moved into the appropriate box from agroup of boxes that are awaiting ordered items for shipment tocustomers. The mug in the tray is identified according to a uniqueidentifier number, such as by a scanned barcode or radio frequencyidentification tag. The unique identifier of the mug is used to access arecord about the mug from an item database to determine a stored weightand a stored digital model representing the shape of the mug. A cameraor other optical imaging device scans the mug, providing informationabout the mug's orientation, e.g., in this case that the mug is on itsside with the open top facing the camera and the handle to the right.The weight, shape, and orientation of the mug are collectively used withany other relevant and/or available information (e.g., about the sizeand orientation of the box into which the mug is to be placed to fulfillthe order) to query a grasping strategy database for a grasping strategyappropriate for this situation. Assuming multiple strategies areavailable, the highest ranked strategy is accessed, which in this casecauses the robotic arm to use a vacuum end effector (rather than anelectromechanical pincher that is also available), approach the mug fromthe open top to grasp the mug on the bottom inside of the mug using 60%of the robotic arm's suction capacity, and rotate while moving towardthe target box so that the mug is set down with the bottom facing downand the handle aligned in the foam slot specially formed in the leftside of the box to receive the handle. Alternatively, assuming that nostrategies are available for this situation, new grasping strategies maybe generated. For example, grasping strategies for similar items (suchas, other mugs, other items determined to be similar in shape to mugs,etc.) may be accessed and/or adapted to provide a grasping strategy forthe mug presented to the robotic arm. As another example, the humanoperator may provide input about how the mug may be effectively graspedby the robotic arm, such as by selecting from different optionspresented on a screen or by donning a glove and grasping the mug so thata grasping strategy for the robotic arm may be generated usinginformation from features on the glove (e.g., pressure sensors, tactilesensors, or fiducial markers used to track the motion of the glove withan optical imaging device). After the robotic arm has performed aninstructed grasping strategy in an attempt to grasp the mug, a rankingfor the instructed strategy can be determined or updated, such as basedon an evaluation of the grasping strategy according to differentcriteria (e.g., whether the mug was successfully transferred from thetray and/or to the target box, an amount of time elapsed to move the mugusing the grasping strategy, and/or whether the mug was damaged duringtransfer).

Referring now to the drawings, in which like reference numerals and/ornames may refer to like elements, FIG. 1 illustrates an inventory system10 having a robotic arm or manipulator 12 configured to grasp inventoryitems 40. Although the description herein primarily refers to a roboticarm 12, any other mechatronic or robotic device may be used in lieu ofor in addition to an arm. Additionally, the terms “grasping,”“gripping,” or the like as used herein should be understood to includeany physical manipulation of objects, including, but not limited to,picking up, pushing, pulling, compressing, stretching, and moving. Thesystem 10 may include the robotic arm 12, a grasping environment 14, aninventory item 40, a sensor package 16, a controller 32, an itemgripping database 36, an item database 37, a human input device 38, anda human operator 34. The item database 37 and the item gripping database36, although depicted as separate in FIG. 1, may share structure and/orcontent. The grasping environment 14, which is depicted in FIG. 1 as aplatform, such as a table or stand, may correspond to any structure orenvironment maintaining an item (such as inventory item 40) that is tobe grasped by the robotic arm 12. For example, the grasping environment14 may equally correspond to a bin of an inventory holder, a drawer, atray, or other structures associated with inventory systems disclosedherein.

The sensor package 16 includes one or more sensors (of like or varyingtype) arranged to detect the item 40 while the item 40 is beingmaintained by the grasping environment 14. The sensor package 16communicates detected attributes (as at 46), such as weight, geometriccharacteristics (e.g., size, position, or orientation), electricalconductivity, magnetic properties, surface characteristics (e.g., howslippery or porous the item is), deformability, and/or structuralintegrity of the item 40, to the controller 32. The detected attributesmay also include a unique identifier of the item 40, such as a barcode-or RFID-encoded serial number. Based on the detected attributes, thecontroller 32 may access (as at 49) the item database 37, such as toaccess a record for the inventory item 40. The record can includeinformation about attributes of the item, such as weight, size, shape,or other physical characteristics of the item. Based on the record fromthe item database 37 and/or the detected attributes from the sensorpackage 16, the controller 32 may access (as at 48) an item grippingdatabase 36 to access an item grasping strategy stored for that item oritems with similar characteristics. The controller 32 can provideinstructions to the robotic arm 12 for gripping the item 40 based on thegripping strategy accessed from the gripping database at 36 (e.g., at52).

In some scenarios—which may include when a grasping strategy from theitem gripping database 36 is not available or not complete—human inputfor a grasping strategy can be received by the controller 32, as at 54.To this end, the human operator 34 may provide a suggestion regardinghow the inventory item 40 should be grasped by the robotic arm 12, andmake the suggestion via the human input device 38. As non-limitingexamples, the human input device 38 may include a computer interface bywhich the human operator 34 can input instructions, may observe a humanaction for grasping an item to learn and/or determine information forforming a grasping strategy, and/or may provide a virtual environment inwhich the human operator can perform a simulated grasping of the item toobtain information for learning and/or determining a grasping strategy.The grasping strategy instructed by the controller 32 to the robotic arm12 (e.g., at 52) can be based on a combination of the human input forgrasping strategy communicated at 54, the detected attributescommunicated at 46, the record accessed from the item database 37 at 49,and/or the information accessed from the item gripping database 36 at48. The controller 32 may update the item gripping database 36 (e.g., at48) and/or the item database 37 based on the human input graspingstrategy received at 54 and/or detected attributes communicated at 46,either of which may include feedback about the success of the graspingstrategy implemented. Such updating and accessing of the item database37 and/or the item gripping database 36 can allow robotic arms 12throughout the inventory system 10 (and throughout other inventorysystems having access to the item gripping database 36) to be used toeffectively move inventory items 40 between elements within theinventory system so as to increase efficiency and throughput.

FIG. 2 illustrates the contents of an inventory system 10 according tosome embodiments of the present disclosure. Inventory system 10 includesa management module 15, one or more mobile drive units 20, one or moreinventory holders 30, and one or more inventory stations 50. Mobiledrive units 20 transport inventory holders 30 between points within aworkspace 70 in response to commands communicated by management module15. Each inventory holder 30 stores one or more types of inventoryitems. As a result, inventory system 10 is capable of moving inventoryitems between locations within workspace 70 to facilitate the entry,processing, and/or removal of inventory items from inventory system 10and the completion of other tasks involving inventory items.

Management module 15 assigns tasks to appropriate components ofinventory system 10 and coordinates operation of the various componentsin completing the tasks. These tasks may relate not only to the movementand processing of inventory items, but also to the management andmaintenance of the components of inventory system 10. For example,management module 15 may assign portions of workspace 70 as parkingspaces for mobile drive units 20, the scheduled recharge or replacementof mobile drive unit batteries, the storage of empty inventory holders30, or any other operations associated with the functionality supportedby inventory system 10 and its various components. Management module 15may select components of inventory system 10 to perform these tasks andcommunicate appropriate commands and/or data to the selected componentsto facilitate completion of these operations. Although shown in FIG. 2as a single, discrete component, management module 15 may representmultiple components and may represent or include portions of mobiledrive units 20 or other elements of inventory system 10. As a result,any or all of the interaction between a particular mobile drive unit 20and management module 15 that is described below may, in particularembodiments, represent peer-to-peer communication between that mobiledrive unit 20 and one or more other mobile drive units 20. The contentsand operation of an example embodiment of management module 15 arediscussed further below with respect to FIG. 3.

Mobile drive units 20 move inventory holders 30 between locations withinworkspace 70. Mobile drive units 20 may represent any devices orcomponents appropriate for use in inventory system 10 based on thecharacteristics and configuration of inventory holders 30 and/or otherelements of inventory system 10. In a particular embodiment of inventorysystem 10, mobile drive units 20 represent independent, self-powereddevices configured to freely move about workspace 70. Examples of suchinventory systems are disclosed in U.S. Patent Publication No.2012/0143427, published on Jun. 7, 2012, titled “SYSTEM AND METHOD FORPOSITIONING A MOBILE DRIVE UNIT” and U.S. Pat. No. 8,280,547, issued onOct. 2, 2012, titled “METHOD AND SYSTEM FOR TRANSPORTING INVENTORYITEMS”, the entire disclosures of which are herein incorporated byreference. In alternative embodiments, mobile drive units 20 representelements of a tracked inventory system configured to move inventoryholder 30 along tracks, rails, cables, crane system, or other guidanceor support elements traversing workspace 70. In such an embodiment,mobile drive units 20 may receive power and/or support through aconnection to the guidance elements, such as a powered rail.Additionally, in particular embodiments of inventory system 10 mobiledrive units 20 may be configured to utilize alternative conveyanceequipment to move within workspace 70 and/or between separate portionsof workspace 70. The contents and operation of an example embodiment ofa mobile drive unit 20 are discussed further below with respect to FIGS.4 and 5.

Additionally, mobile drive units 20 may be capable of communicating withmanagement module 15 to receive information identifying selectedinventory holders 30, transmit the locations of mobile drive units 20,or exchange any other suitable information to be used by managementmodule 15 or mobile drive units 20 during operation. Mobile drive units20 may communicate with management module 15 wirelessly, using wiredconnections between mobile drive units 20 and management module 15,and/or in any other appropriate manner. As one example, particularembodiments of mobile drive unit 20 may communicate with managementmodule 15 and/or with one another using 802.11, Bluetooth, or InfraredData Association (IrDA) standards, or any other appropriate wirelesscommunication protocol. As another example, in a tracked inventorysystem 10, tracks or other guidance elements upon which mobile driveunits 20 move may be wired to facilitate communication between mobiledrive units 20 and other components of inventory system 10. Furthermore,as noted above, management module 15 may include components ofindividual mobile drive units 20. Thus, for the purposes of thisdescription and the claims that follow, communication between managementmodule 15 and a particular mobile drive unit 20 may representcommunication between components of a particular mobile drive unit 20.In general, mobile drive units 20 may be powered, propelled, andcontrolled in any manner appropriate based on the configuration andcharacteristics of inventory system 10.

Inventory holders 30 store inventory items. In a particular embodiment,inventory holders 30 include multiple storage bins with each storage bincapable of holding one or more types of inventory items. Inventoryholders 30 are capable of being carried, rolled, and/or otherwise movedby mobile drive units 20. In particular embodiments, inventory holder 30may provide additional propulsion to supplement that provided by mobiledrive unit 20 when moving inventory holder 30.

Additionally, in particular embodiments, inventory items 40 may alsohang from hooks or bars (not shown) within or on inventory holder 30. Ingeneral, inventory holder 30 may store inventory items 40 in anyappropriate manner within inventory holder 30 and/or on the externalsurface of inventory holder 30.

Additionally, each inventory holder 30 may include a plurality of faces,and each bin may be accessible through one or more faces of theinventory holder 30. For example, in a particular embodiment, inventoryholder 30 includes four faces. In such an embodiment, bins located at acorner of two faces may be accessible through either of those two faces,while each of the other bins is accessible through an opening in one ofthe four faces. Mobile drive unit 20 may be configured to rotateinventory holder 30 at appropriate times to present a particular faceand the bins associated with that face to an operator or othercomponents of inventory system 10.

Inventory items represent any objects suitable for storage, retrieval,and/or processing in an automated inventory system 10. For the purposesof this description, “inventory items” may represent any one or moreobjects of a particular type that are stored in inventory system 10.Thus, a particular inventory holder 30 is currently “storing” aparticular inventory item if the inventory holder 30 currently holds oneor more units of that type. As one example, inventory system 10 mayrepresent a mail order warehouse facility, and inventory items mayrepresent merchandise stored in the warehouse facility. Duringoperation, mobile drive units 20 may retrieve inventory holders 30containing one or more inventory items requested in an order to bepacked for delivery to a customer or inventory holders 30 carryingpallets containing aggregated collections of inventory items forshipment. Moreover, in particular embodiments of inventory system 10,boxes containing completed orders may themselves represent inventoryitems.

In particular embodiments, inventory system 10 may also include one ormore inventory stations 50. Inventory stations 50 represent locationsdesignated for the completion of particular tasks involving inventoryitems. Such tasks may include the removal of inventory items frominventory holders 30, the introduction of inventory items into inventoryholders 30, the counting of inventory items in inventory holders 30, thedecomposition of inventory items (e.g., from pallet- or case-sizedgroups to individual inventory items), the consolidation of inventoryitems between inventory holders 30, and/or the processing or handling ofinventory items in any other suitable manner. In particular embodiments,inventory stations 50 may just represent the physical locations where aparticular task involving inventory items can be completed withinworkspace 70. In alternative embodiments, inventory stations 50 mayrepresent both the physical location and also any appropriate equipmentfor processing or handling inventory items, such as scanners formonitoring the flow of inventory items in and out of inventory system10, communication interfaces for communicating with management module15, and/or any other suitable components. Inventory stations 50 may becontrolled, entirely or in part, by human operators or may be fullyautomated. Moreover, the human or automated operators of inventorystations 50 may be capable of performing certain tasks to inventoryitems, such as packing, counting, or transferring inventory items, aspart of the operation of inventory system 10.

Workspace 70 represents an area associated with inventory system 10 inwhich mobile drive units 20 can move and/or inventory holders 30 can bestored. For example, workspace 70 may represent all or part of the floorof a mail-order warehouse in which inventory system 10 operates.Although FIG. 2 shows, for the purposes of illustration, an embodimentof inventory system 10 in which workspace 70 includes a fixed,predetermined, and finite physical space, particular embodiments ofinventory system 10 may include mobile drive units 20 and inventoryholders 30 that are configured to operate within a workspace 70 that isof variable dimensions and/or an arbitrary geometry. While FIG. 2illustrates a particular embodiment of inventory system 10 in whichworkspace 70 is entirely enclosed in a building, alternative embodimentsmay utilize workspaces 70 in which some or all of the workspace 70 islocated outdoors, within a vehicle (such as a cargo ship), or otherwiseunconstrained by any fixed structure.

In operation, management module 15 selects appropriate components tocomplete particular tasks and transmits task assignments 18 to theselected components to trigger completion of the relevant tasks. Eachtask assignment 18 defines one or more tasks to be completed by aparticular component. These tasks may relate to the retrieval, storage,replenishment, and counting of inventory items and/or the management ofmobile drive units 20, inventory holders 30, inventory stations 50 andother components of inventory system 10. Depending on the component andthe task to be completed, a particular task assignment 18 may identifylocations, components, and/or actions associated with the correspondingtask and/or any other appropriate information to be used by the relevantcomponent in completing the assigned task.

In particular embodiments, management module 15 generates taskassignments 18 based, in part, on inventory requests that managementmodule 15 receives from other components of inventory system 10 and/orfrom external components in communication with management module 15.These inventory requests identify particular operations to be completedinvolving inventory items stored or to be stored within inventory system10 and may represent communication of any suitable form. For example, inparticular embodiments, an inventory request may represent a shippingorder specifying particular inventory items that have been purchased bya customer and that are to be retrieved from inventory system 10 forshipment to the customer. Management module 15 may also generate taskassignments 18 independently of such inventory requests, as part of theoverall management and maintenance of inventory system 10. For example,management module 15 may generate task assignments 18 in response to theoccurrence of a particular event (e.g., in response to a mobile driveunit 20 requesting a space to park), according to a predeterminedschedule (e.g., as part of a daily start-up routine), or at anyappropriate time based on the configuration and characteristics ofinventory system 10. After generating one or more task assignments 18,management module 15 transmits the generated task assignments 18 toappropriate components for completion of the corresponding task. Therelevant components then execute their assigned tasks.

With respect to mobile drive units 20 specifically, management module 15may, in particular embodiments, communicate task assignments 18 toselected mobile drive units 20 that identify one or more destinationsfor the selected mobile drive units 20. Management module 15 may selecta mobile drive unit 20 to assign the relevant task based on the locationor state of the selected mobile drive unit 20, an indication that theselected mobile drive unit 20 has completed a previously-assigned task,a predetermined schedule, and/or any other suitable consideration. Thesedestinations may be associated with an inventory request the managementmodule 15 is executing or a management objective the management module15 is attempting to fulfill. For example, the task assignment may definethe location of an inventory holder 30 to be retrieved, an inventorystation 50 to be visited, a storage location where the mobile drive unit20 should park until receiving another task, or a location associatedwith any other task appropriate based on the configuration,characteristics, and/or state of inventory system 10, as a whole, orindividual components of inventory system 10. For example, in particularembodiments, such decisions may be based on the popularity of particularinventory items, the staffing of a particular inventory station 50, thetasks currently assigned to a particular mobile drive unit 20, and/orany other appropriate considerations.

As part of completing these tasks mobile drive units 20 may dock withand transport inventory holders 30 within workspace 70. Mobile driveunits 20 may dock with inventory holders 30 by connecting to, lifting,and/or otherwise interacting with inventory holders 30 in any othersuitable manner so that, when docked, mobile drive units 20 are coupledto and/or support inventory holders 30 and can move inventory holders 30within workspace 70. While the description below focuses on particularembodiments of mobile drive unit 20 and inventory holder 30 that areconfigured to dock in a particular manner, alternative embodiments ofmobile drive unit 20 and inventory holder 30 may be configured to dockin any manner suitable to allow mobile drive unit 20 to move inventoryholder 30 within workspace 70. Additionally, as noted below, inparticular embodiments, mobile drive units 20 represent all or portionsof inventory holders 30. In such embodiments, mobile drive units 20 maynot dock with inventory holders 30 before transporting inventory holders30 and/or mobile drive units 20 may each remain continually docked witha particular inventory holder 30.

While the appropriate components of inventory system 10 completeassigned tasks, management module 15 may interact with the relevantcomponents to ensure the efficient use of space, equipment, manpower,and other resources available to inventory system 10. As one specificexample of such interaction, management module 15 is responsible, inparticular embodiments, for planning the paths mobile drive units 20take when moving within workspace 70 and for allocating use of aparticular portion of workspace 70 to a particular mobile drive unit 20for purposes of completing an assigned task. In such embodiments, mobiledrive units 20 may, in response to being assigned a task, request a pathto a particular destination associated with the task. Moreover, whilethe description below focuses on one or more embodiments in which mobiledrive unit 20 requests paths from management module 15, mobile driveunit 20 may, in alternative embodiments, generate its own paths.

Components of inventory system 10 may provide information to managementmodule 15 regarding their current state, other components of inventorysystem 10 with which they are interacting, and/or other conditionsrelevant to the operation of inventory system 10. This may allowmanagement module 15 to utilize feedback from the relevant components toupdate algorithm parameters, adjust policies, or otherwise modify itsdecision-making to respond to changes in operating conditions or theoccurrence of particular events.

In addition, while management module 15 may be configured to managevarious aspects of the operation of the components of inventory system10, in particular embodiments, the components themselves may also beresponsible for decision-making relating to certain aspects of theiroperation, thereby reducing the processing load on management module 15.

Thus, based on its knowledge of the location, current state, and/orother characteristics of the various components of inventory system 10and an awareness of all the tasks currently being completed, managementmodule 15 can generate tasks, allot usage of system resources, andotherwise direct the completion of tasks by the individual components ina manner that optimizes operation from a system-wide perspective.Moreover, by relying on a combination of both centralized, system-widemanagement and localized, component-specific decision-making, particularembodiments of inventory system 10 may be able to support a number oftechniques for efficiently executing various aspects of the operation ofinventory system 10. As a result, particular embodiments of managementmodule 15 may, by implementing one or more management techniquesdescribed below, enhance the efficiency of inventory system 10 and/orprovide other operational benefits.

FIG. 3 illustrates in greater detail the components of a particularembodiment of management module 15. As shown, the example embodimentincludes a resource scheduling module 92, a route planning module 94, asegment reservation module 96, an inventory module 97, a communicationinterface module 98, a processor 90, and a memory 91. Management module15 may represent a single component, multiple components located at acentral location within inventory system 10, or multiple componentsdistributed throughout inventory system 10. For example, managementmodule 15 may represent components of one or more mobile drive units 20that are capable of communicating information between the mobile driveunits 20 and coordinating the movement of mobile drive units 20 withinworkspace 70. In general, management module 15 may include anyappropriate combination of hardware and/or software suitable to providethe described functionality.

Processor 90 is operable to execute instructions associated with thefunctionality provided by management module 15. Processor 90 maycomprise one or more general purpose computers, dedicatedmicroprocessors, or other processing devices capable of communicatingelectronic information. Examples of processor 90 include one or moreapplication-specific integrated circuits (ASICs), field programmablegate arrays (FPGAs), digital signal processors (DSPs) and any othersuitable specific or general purpose processors.

Memory 91 stores processor instructions, inventory requests, reservationinformation, state information for the various components of inventorysystem 10 and/or any other appropriate values, parameters, orinformation utilized by management module 15 during operation. Memory 91may represent any collection and arrangement of volatile or nonvolatile,local or remote devices suitable for storing data. Examples of memory 91include, but are not limited to, random access memory (RAM) devices,read only memory (ROM) devices, magnetic storage devices, opticalstorage devices or any other suitable data storage devices.

Resource scheduling module 92 processes received inventory requests andgenerates one or more assigned tasks to be completed by the componentsof inventory system 10. Resource scheduling module 92 may also selectone or more appropriate components for completing the assigned tasksand, using communication interface module 98, communicate the assignedtasks to the relevant components. Additionally, resource schedulingmodule 92 may also be responsible for generating assigned tasksassociated with various management operations, such as prompting mobiledrive units 20 to recharge batteries or have batteries replaced,instructing inactive mobile drive units 20 to park in a location outsidethe anticipated traffic flow or a location near the anticipated site offuture tasks, and/or directing mobile drive units 20 selected for repairor maintenance to move towards a designated maintenance station.

Route planning module 94 receives route requests from mobile drive units20. These route requests identify one or more destinations associatedwith a task the requesting mobile drive unit 20 is executing. Inresponse to receiving a route request, route planning module 94generates a path to one or more destinations identified in the routerequest. Route planning module 94 may implement any appropriatealgorithms utilizing any appropriate parameters, factors, and/orconsiderations to determine the appropriate path. After generating anappropriate path, route planning module 94 transmits a route responseidentifying the generated path to the requesting mobile drive unit 20using communication interface module 98.

Segment reservation module 96 receives reservation requests from mobiledrive units 20 attempting to move along paths generated by routeplanning module 94. These reservation requests request the use of aparticular portion of workspace 70 (referred to herein as a “segment”)to allow the requesting mobile drive unit 20 to avoid collisions withother mobile drive units 20 while moving across the reserved segment. Inresponse to received reservation requests, segment reservation module 96transmits a reservation response granting or denying the reservationrequest to the requesting mobile drive unit 20 using the communicationinterface module 98.

The inventory module 97 maintains information about the location andnumber of inventory items 40 in the inventory system 10. Information canbe maintained about the number of inventory items 40 in a particularinventory holder 30, and the maintained information can include thelocation of those inventory items 40 in the inventory holder 30. Theinventory module 97 can also communicate with the mobile drive units 20,utilizing task assignments 18 to maintain, replenish or move inventoryitems 40 within the inventory system 10.

Communication interface module 98 facilitates communication betweenmanagement module 15 and other components of inventory system 10,including reservation responses, reservation requests, route requests,route responses, and task assignments. These reservation responses,reservation requests, route requests, route responses, and taskassignments may represent communication of any form appropriate based onthe capabilities of management module 15 and may include any suitableinformation. Depending on the configuration of management module 15,communication interface module 98 may be responsible for facilitatingeither or both of wired and wireless communication between managementmodule 15 and the various components of inventory system 10. Inparticular embodiments, management module 15 may communicate usingcommunication protocols such as 802.11, Bluetooth, or Infrared DataAssociation (IrDA) standards. Furthermore, management module 15 may, inparticular embodiments, represent a portion of mobile drive unit 20 orother components of inventory system 10. In such embodiments,communication interface module 98 may facilitate communication betweenmanagement module 15 and other parts of the same system component.

In general, resource scheduling module 92, route planning module 94,segment reservation module 96, inventory module 97, and communicationinterface module 98 may each represent any appropriate hardware and/orsoftware suitable to provide the described functionality. In addition,as noted above, management module 15 may, in particular embodiments,represent multiple different discrete components and any or all ofresource scheduling module 92, route planning module 94, segmentreservation module 96, inventory module 97, and communication interfacemodule 98 may represent components physically separate from theremaining elements of management module 15. Moreover, any two or more ofresource scheduling module 92, route planning module 94, segmentreservation module 96, inventory module 97, and communication interfacemodule 98 may share common components. For example, in particularembodiments, resource scheduling module 92, route planning module 94,segment reservation module 96, and inventory module 97 representcomputer processes executing on processor 90 and communication interfacemodule 98 comprises a wireless transmitter, a wireless receiver, and arelated computer process executing on processor 90.

FIGS. 4 and 5 illustrate in greater detail the components of aparticular embodiment of mobile drive unit 20. In particular, FIGS. 4and 5 include a front and side view of an example mobile drive unit 20.Mobile drive unit 20 includes a docking head 110, a drive module 120, adocking actuator 130, and a control module 170. Additionally, mobiledrive unit 20 may include one or more sensors configured to detect ordetermine the location of mobile drive unit 20, inventory holder 30,and/or other appropriate elements of inventory system 10. In theillustrated embodiment, mobile drive unit 20 includes a position sensor140, a holder sensor 150, an obstacle sensor 160, and an identificationsignal transmitter 162.

Docking head 110, in particular embodiments of mobile drive unit 20,couples mobile drive unit 20 to inventory holder 30 and/or supportsinventory holder 30 when mobile drive unit 20 is docked to inventoryholder 30. Docking head 110 may additionally allow mobile drive unit 20to maneuver inventory holder 30, such as by lifting inventory holder 30,propelling inventory holder 30, rotating inventory holder 30, and/ormoving inventory holder 30 in any other appropriate manner. Docking head110 may also include any appropriate combination of components, such asribs, spikes, and/or corrugations, to facilitate such manipulation ofinventory holder 30. For example, in particular embodiments, dockinghead 110 may include a high-friction portion that abuts a portion ofinventory holder 30 while mobile drive unit 20 is docked to inventoryholder 30. In such embodiments, frictional forces created between thehigh-friction portion of docking head 110 and a surface of inventoryholder 30 may induce translational and rotational movement in inventoryholder 30 when docking head 110 moves and rotates, respectively. As aresult, mobile drive unit 20 may be able to manipulate inventory holder30 by moving or rotating docking head 110, either independently or as apart of the movement of mobile drive unit 20 as a whole.

Drive module 120 propels mobile drive unit 20 and, when mobile driveunit 20 and inventory holder 30 are docked, inventory holder 30. Drivemodule 120 may represent any appropriate collection of componentsoperable to propel mobile drive unit 20. For example, in the illustratedembodiment, drive module 120 includes motorized axle 122, a pair ofmotorized wheels 124, and a pair of stabilizing wheels 126. Onemotorized wheel 124 is located at each end of motorized axle 122, andone stabilizing wheel 126 is positioned at each end of mobile drive unit20.

Docking actuator 130 moves docking head 110 towards inventory holder 30to facilitate docking of mobile drive unit 20 and inventory holder 30.Docking actuator 130 may also be capable of adjusting the position ororientation of docking head 110 in other suitable manners to facilitatedocking. Docking actuator 130 may include any appropriate components,based on the configuration of mobile drive unit 20 and inventory holder30, for moving docking head 110 or otherwise adjusting the position ororientation of docking head 110. For example, in the illustratedembodiment, docking actuator 130 includes a motorized shaft (not shown)attached to the center of docking head 110. The motorized shaft isoperable to lift docking head 110 as appropriate for docking withinventory holder 30.

Drive module 120 may be configured to propel mobile drive unit 20 in anyappropriate manner. For example, in the illustrated embodiment,motorized wheels 124 are operable to rotate in a first direction topropel mobile drive unit 20 in a forward direction. Motorized wheels 124are also operable to rotate in a second direction to propel mobile driveunit 20 in a backward direction. In the illustrated embodiment, drivemodule 120 is also configured to rotate mobile drive unit 20 by rotatingmotorized wheels 124 in different directions from one another or byrotating motorized wheels 124 at different speeds from one another.

Position sensor 140 represents one or more sensors, detectors, or othercomponents suitable for determining the location of mobile drive unit 20in any appropriate manner. For example, in particular embodiments, theworkspace 70 associated with inventory system 10 includes a number offiducial marks that mark points on a two-dimensional grid that coversall or a portion of workspace 70. In such embodiments, position sensor140 may include a camera and suitable image- and/or video-processingcomponents, such as an appropriately-programmed digital signalprocessor, to allow position sensor 140 to detect fiducial marks withinthe camera's field of view. Control module 170 may store locationinformation that position sensor 140 updates as position sensor 140detects fiducial marks. As a result, position sensor 140 may utilizefiducial marks to maintain an accurate indication of the location mobiledrive unit 20 and to aid in navigation when moving within workspace 70.

Holder sensor 150 represents one or more sensors, detectors, or othercomponents suitable for detecting inventory holder 30 and/ordetermining, in any appropriate manner, the location of inventory holder30, as an absolute location or as a position relative to mobile driveunit 20. Holder sensor 150 may be capable of detecting the location of aparticular portion of inventory holder 30 or inventory holder 30 as awhole. Mobile drive unit 20 may then use the detected information fordocking with or otherwise interacting with inventory holder 30.

Obstacle sensor 160 represents one or more sensors capable of detectingobjects located in one or more different directions in which mobiledrive unit 20 is capable of moving. Obstacle sensor 160 may utilize anyappropriate components and techniques, including optical, radar, sonar,pressure-sensing and/or other types of detection devices appropriate todetect objects located in the direction of travel of mobile drive unit20. In particular embodiments, obstacle sensor 160 may transmitinformation describing objects it detects to control module 170 to beused by control module 170 to identify obstacles and to take appropriateremedial actions to prevent mobile drive unit 20 from colliding withobstacles and/or other objects.

Obstacle sensor 160 may also detect signals transmitted by other mobiledrive units 20 operating in the vicinity of the illustrated mobile driveunit 20. For example, in particular embodiments of inventory system 10,one or more mobile drive units 20 may include an identification signaltransmitter 162 that transmits a drive identification signal. The driveidentification signal indicates to other mobile drive units 20 that theobject transmitting the drive identification signal is in fact a mobiledrive unit. Identification signal transmitter 162 may be capable oftransmitting infrared, ultraviolet, audio, visible light, radio, and/orother suitable signals that indicate to recipients that the transmittingdevice is a mobile drive unit 20.

Additionally, in particular embodiments, obstacle sensor 160 may also becapable of detecting state information transmitted by other mobile driveunits 20. For example, in particular embodiments, identification signaltransmitter 162 may be capable of including state information relatingto mobile drive unit 20 in the transmitted identification signal. Thisstate information may include, but is not limited to, the position,velocity, direction, and the braking capabilities of the transmittingmobile drive unit 20. In particular embodiments, mobile drive unit 20may use the state information transmitted by other mobile drive units toavoid collisions when operating in close proximity with those othermobile drive units.

Control module 170 monitors and/or controls operation of drive module120 and docking actuator 130. Control module 170 may also receiveinformation from sensors such as position sensor 140 and holder sensor150 and adjust the operation of drive module 120, docking actuator 130,and/or other components of mobile drive unit 20 based on thisinformation. Additionally, in particular embodiments, mobile drive unit20 may be configured to communicate with a management device ofinventory system 10 and control module 170 may receive commandstransmitted to mobile drive unit 20 and communicate information back tothe management device utilizing appropriate communication components ofmobile drive unit 20. Control module 170 may include any appropriatehardware and/or software suitable to provide the describedfunctionality. In particular embodiments, control module 170 includes ageneral-purpose microprocessor programmed to provide the describedfunctionality. Additionally, control module 170 may include all orportions of docking actuator 130, drive module 120, position sensor 140,and/or holder sensor 150, and/or share components with any of theseelements of mobile drive unit 20.

Moreover, in particular embodiments, control module 170 may includehardware and software located in components that are physically distinctfrom the device that houses drive module 120, docking actuator 130,and/or the other components of mobile drive unit 20 described above. Forexample, in particular embodiments, each mobile drive unit 20 operatingin inventory system 10 may be associated with a software process(referred to here as a “drive agent”) operating on a server that is incommunication with the device that houses drive module 120, dockingactuator 130, and other appropriate components of mobile drive unit 20.This drive agent may be responsible for requesting and receiving tasks,requesting and receiving routes, transmitting state informationassociated with mobile drive unit 20, and/or otherwise interacting withmanagement module 15 and other components of inventory system 10 onbehalf of the device that physically houses drive module 120, dockingactuator 130, and the other appropriate components of mobile drive unit20. As a result, for the purposes of this description and the claimsthat follow, the term “mobile drive unit” includes software and/orhardware, such as agent processes, that provides the describedfunctionality on behalf of mobile drive unit 20 but that may be locatedin physically distinct devices from the drive module 120, dockingactuator 130, and/or the other components of mobile drive unit 20described above.

While FIGS. 4 and 5 illustrate a particular embodiment of mobile driveunit 20 containing certain components and configured to operate in aparticular manner, mobile drive unit 20 may represent any appropriatecomponent and/or collection of components configured to transport and/orfacilitate the transport of inventory holders 30. As another example,mobile drive unit 20 may represent part of an overhead crane system inwhich one or more crane assemblies are capable of moving within anetwork of wires or rails to a position suitable to dock with aparticular inventory holder 30. After docking with inventory holder 30,the crane assembly may then lift inventory holder 30 and move inventoryto another location for purposes of completing an assigned task.

Furthermore, in particular embodiments, mobile drive unit 20 mayrepresent all or a portion of inventory holder 30. Inventory holder 30may include motorized wheels or any other components suitable to allowinventory holder 30 to propel itself. As one specific example, a portionof inventory holder 30 may be responsive to magnetic fields. Inventorysystem 10 may be able to generate one or more controlled magnetic fieldscapable of propelling, maneuvering and/or otherwise controlling theposition of inventory holder 30 as a result of the responsive portion ofinventory holder 30. In such embodiments, mobile drive unit 20 mayrepresent the responsive portion of inventory holder 30 and/or thecomponents of inventory system 10 responsible for generating andcontrolling these magnetic fields. While this description providesseveral specific examples, mobile drive unit 20 may, in general,represent any appropriate component and/or collection of componentsconfigured to transport and/or facilitate the transport of inventoryholders 30.

FIG. 6 illustrates in greater detail the components of a particularembodiment of inventory holder 30. In particular, FIG. 6 illustrates thestructure and contents of one side of an example inventory holder 30. Ina particular embodiment, inventory holder 30 may comprise any number offaces with similar or different structure. As illustrated, inventoryholder 30 includes a frame 310, a plurality of legs 328, and a dockingsurface 350.

Frame 310 holds inventory items 40. Frame 310 provides storage space forstoring inventory items 40 external or internal to frame 310. Thestorage space provided by frame 310 may be divided into a plurality ofinventory bins 320, each capable of holding inventory items 40.Inventory bins 320 may include any appropriate storage elements, such asbins, compartments, or hooks.

In a particular embodiment, frame 310 is composed of a plurality oftrays 322 stacked upon one another and attached to or stacked on a base318. In such an embodiment, inventory bins 320 may be formed by aplurality of adjustable dividers 324 that may be moved to resize one ormore inventory bins 320. In alternative embodiments, frame 310 mayrepresent a single inventory bin 320 that includes a single tray 322 andno adjustable dividers 324. Additionally, in particular embodiments,frame 310 may represent a load-bearing surface mounted on mobilityelement 330. Inventory items 40 may be stored on such an inventoryholder 30 by being placed on frame 310. In general, frame 310 mayinclude storage internal and/or external storage space divided into anyappropriate number of inventory bins 320 in any appropriate manner.

Additionally, in a particular embodiment, frame 310 may include aplurality of device openings 326 that allow mobile drive unit 20 toposition docking head 110 adjacent docking surface 350. The size, shape,and placement of device openings 326 may be determined based on thesize, the shape, and other characteristics of the particular embodimentof mobile drive unit 20 and/or inventory holder 30 utilized by inventorysystem 10. For example, in the illustrated embodiment, frame 310includes four legs 328 that form device openings 326 and allow mobiledrive unit 20 to position mobile drive unit 20 under frame 310 andadjacent to docking surface 350. The length of legs 328 may bedetermined based on a height of mobile drive unit 20.

Docking surface 350 comprises a portion of inventory holder 30 thatcouples to, abuts, and/or rests upon a portion of docking head 110, whenmobile drive unit 20 is docked to inventory holder 30. Additionally,docking surface 350 supports a portion or all of the weight of inventoryholder 30 while inventory holder 30 is docked with mobile drive unit 20.The composition, shape, and/or texture of docking surface 350 may bedesigned to facilitate maneuvering of inventory holder 30 by mobiledrive unit 20. For example, as noted above, in particular embodiments,docking surface 350 may comprise a high-friction portion. When mobiledrive unit 20 and inventory holder 30 are docked, frictional forcesinduced between docking head 110 and this high-friction portion mayallow mobile drive unit 20 to maneuver inventory holder 30.Additionally, in particular embodiments, docking surface 350 may includeappropriate components suitable to receive a portion of docking head110, couple inventory holder 30 to mobile drive unit 20, and/orfacilitate control of inventory holder 30 by mobile drive unit 20.

Holder identifier 360 marks a predetermined portion of inventory holder30 and mobile drive unit 20 may use holder identifier 360 to align withinventory holder 30 during docking and/or to determine the location ofinventory holder 30. More specifically, in particular embodiments,mobile drive unit 20 may be equipped with components, such as holdersensor 150, that can detect holder identifier 360 and determine itslocation relative to mobile drive unit 20. As a result, mobile driveunit 20 may be able to determine the location of inventory holder 30 asa whole. For example, in particular embodiments, holder identifier 360may represent a reflective marker that is positioned at a predeterminedlocation on inventory holder 30 and that holder sensor 150 can opticallydetect using an appropriately-configured camera.

Depending on the configuration and characteristics of mobile drive unit20 and inventory system 10, mobile drive unit 20 may move inventoryholder 30 using a variety of appropriate methods. In a particularembodiment, mobile drive unit 20 is capable of moving inventory holder30 along a two-dimensional grid, combining movement along straight-linesegments with ninety-degree rotations and arcing paths to transportinventory holder 30 from the first location to the second location.Additionally, while moving, mobile drive unit 20 may use fixed objectslocated in the workspace as reference points to assist in navigation.For example, in particular embodiments, inventory system 10 includesmultiple fiducial marks. Mobile drive unit 20 may be configured todetect fiducial marks and to determine the location of mobile drive unit20 and/or measure its movement based on the detection of fiducial marks.

After mobile drive unit 20 arrives at the second location, mobile driveunit 20 may perform appropriate operations to facilitate access toinventory items 40 stored in inventory holder 30. For example, mobiledrive unit 20 may rotate inventory holder 30 to present a particularface of inventory holder 30 to an operator of inventory system 10 orother suitable party, such as a packer selecting inventory items 40 frominventory holder 30. Mobile drive unit 20 may also undock from inventoryholder 30. Alternatively, instead of undocking at the second location,mobile drive unit 20 may transport inventory holder 30 back to the firstlocation or to a third location after any appropriate actions have beentaken involving inventory items 40. For example, after a packer hasremoved particular inventory items 40 from inventory holder 30, mobiledrive unit 20 may return inventory holder 30 to its original storagelocation, a new storage location, or another inventory station. Mobiledrive unit 20 may then undock from inventory holder 30 at this newlocation.

As described above, embodiments herein are directed to robotic arms ormanipulators 12 (hereinafter “robotic arms”) and associated componentsand techniques for automated grasping of inventory items 40 and/or otherobjects within an inventory system 10. In accordance with someembodiments, the management module 15 is utilized to control operationsof robotic arms 12 and the instructions associated therewith. FIG. 7illustrates in greater detail modules of the management module 15 thatmay be used in association with robotic arms 12. Example modules areshown in FIG. 7, but functions and embodiments described herein canutilize a subset of the features provided by the modules and/oradditional functions can be provided. Additionally, while the examplemodules will now be briefly discussed with regard to FIG. 7, furtherspecific details regarding the example modules are provided below in thedescriptions of subsequent Figures.

As illustrated in FIG. 7, the modules can include an attribute detectionmodule 710, a database query module 715, a human-based grasping strategymodule 720, a constraints module 725, a grasping strategy selectionmodule 730, a grasping strategy instruction module 735, a graspingstrategy evaluation module 740, and a database update module 745. Theattribute detection module 710 receives information from sensors, suchas the sensor package 16 of FIG. 1, and determines attributes ofinventory items 40 detected by the sensors. The database query module715 can access information from the item database 37 and/or the itemgripping database 36 of FIG. 1, such as based on the information fromthe attribute detection module 710 or other modules described herein.The human-based grasping strategy module 720 handles requests andreceipt of any human input for grasping strategy, such as at 54 inFIG. 1. The constraints module 725 can provide information aboutconstraints to available grasping strategies from various sources, suchas information about where an item grasped by a robotic arm 12 is to beplaced. The grasping strategy selection module 730 may select and/ordetermine a grasping strategy to be provided to a robotic arm 12 for aparticular grasping action, such as based on information from othermodules described herein. The grasping strategy instruction module 735can provide the instructions embodying the grasping strategy selected bythe grasping strategy selection module 730 to the robotic arm 12 for aparticular grasping action. The grasping strategy evaluation module 740can receive information about a grasping action performed in response toinstructions provided by the grasping strategy instruction module 735and evaluate a success of the grasping strategy for the item orattributes of the item involved in the particular grasping action. Thedatabase update module 745 may update the item database 37 and/or theitem gripping database 36, such as utilizing information from othermodules described herein.

The attribute detection module 710 can interact with any number and/ortype of sensors to determine attributes of an item to be grasped. Forexample, the attribute detection module 710 can receive information fromimaging devices or optical sensors to determine physicalcharacteristics, such as size, shape, position, orientation, and/orsurface characteristics (e.g., how porous and/or slippery the item isbased on the surface appearance). Any suitable optical technology can beutilized, including, but not limited to, two-dimensional cameras, depthsensors, time of flight sensing (e.g., broadcasting a source of lightand determining a time of reflection for each pixel to determine adistance from the sensor for each pixel to determine a three-dimensionalarray of data points representing a virtual model of the sensed item andenvironment), structured light sensing (e.g., projecting a known imagefrom a light source, observing the image as distorted by variations inthe surface of the detected item, and analyzing the distortions withrespect to the projected image to determine positioning of the featuresthat caused the distortion), stereo sensing (e.g., analyzing differencesin images collected from multiple cameras arranged at known offsets fromone another to generate a point cloud or digital model), active stereosensing (e.g., projecting a pattern of light to improve precision ofdetection of features while using stereo sensing), any otheroptically-based methodology of observing light for generating a digitalrepresentation of a physical object, or any combination thereof.

In some embodiments, the attribute detection module 710 canadditionally, or alternatively, receive active sensing information fromother sensors (e.g., force sensing, tactile sensing, pressure sensing,voltage sensing, conductance sensing, ultrasonic sensing, x-ray sensing,or other sensing), such as to determine physical attributes of adetected item to be grasped or its surroundings, such as structuralintegrity, deformability, weight, surface characteristics (e.g., howslippery the item is), or other physical attributes of a detectedobject.

Information about such physical attributes of an item, as determinedbased on information from sensors, can be useful for a number offunctions. For example, the detected attributes may facilitateidentification of an item as an item or type of item that has previouslybeen identified and included in a database or entry in a database ofitems managed by the inventory system. As an illustrative example,active sensing of deformability may facilitate a determination ofwhether a set of information received from a sensor corresponds to abowling ball or a beach ball, which may otherwise have similarthree-dimensional geometry.

In some embodiments, a unique identifier may be an attribute detected bythe attribute detection module 710. For example, the attribute detectionmodule 710 may determine a UPC or SKU (universal product code or stockkeeping unit) of an inventory item based on a barcode detected by anoptical scanner or from an RFID tag detected by a RFID reader.Determining a unique identifier of an item may permit or facilitatecertain other uses of detected attributes. For example, an item having aunique identifier may have an associated database entry (e.g., from arecord in the item database 37 of FIG. 1) regarding anticipated physicalattributes of the item that may be used to locate and grasp the itemwithin a particular environment (e.g., identifying a shape of the itembased on the unique identifier so that that shape may be located in adetected environment to provide direction to the robotic arm to selectthe correct item out of a group of items). In some embodiments, a uniqueidentifier may provide confirmation that a grasped item is the item thatwas intended to be grasped. In some embodiments, information related tothe unique identifier may be used to determine if an item is damaged.For example, a barcode for an item may be utilized to determine expectedcontours of the item or its packaging (e.g., from a record in the itemdatabase 37 of FIG. 1), which can be compared with detected informationabout the item to determine if the item has been damaged or deformed inan unacceptable manner. In some embodiments, detected attributes aboutan object may be utilized to differentiate or distinguish betweendifferent items or conditions of items having the same identifier. As anillustrative example, detected attributes may indicate whether a coffeemug identified by an RFID reader is wrapped in plastic wrap or in a box,which may impact a grasping strategy for gripping the coffee mug. Asanother illustrative example, a book from one supplier may have the samebarcode as a disc from another supplier, and a camera or other opticalimaging device may provide information about the size of the detectedobject in order to distinguish between the two possibilities.

A database query module 715, as mentioned above, may access informationabout item grasping strategy from a database, such as the item graspingdatabase 36 of FIG. 1. The database query module 715 may additionally oralternatively access records about items from a database, such as fromthe item database 37 of FIG. 1. In some embodiments, the database querymodule 715 may receive grasping strategy information based on a detecteditem being a recognized item (e.g., corresponding to a particular SKU).In some embodiments, the database query module 715 may access graspingstrategy information based on detected physical attributes of the item(e.g., a size, shape, position, orientation, weight, or other attributesof the item regardless of a SKU of the item). The database query module715 may receive information about grasping strategies based on therecognized item and/or characteristics.

A grasping strategy can include any information regarding the manner inwhich a robotic arm 12 is to attempt to grasp a particular item 40 orgroup of items. For example, a grasping strategy may include anindication of how the robotic arm is to approach the item to be grasped,an indication of one or more end effectors to be utilized by the roboticarm, and/or an indication of a level of intensity (e.g., amount offorce, pressure, voltage, current, etc.) with which the robotic arm isto operate the end effector(s). In some embodiments, the graspingstrategy may also include a number of items to be simultaneouslygrasped.

The approach identified in a grasping strategy may include a directionfrom which the robotic arm is to approach the item (e.g., from above,from a side, from an angle) and/or a sequence of motions by which therobotic arm is to perform a particular grasping operation, which mayinclude reaching the target item, grasping the target item, moving thetarget item to a target location, and/or releasing the target item inthe target location.

As to end effectors identified in a grasping strategy, the robotic arm12 may include one or more end effectors and may be capable of utilizingmultiple end effectors in conjunction with one another or asalternatives to one another. As illustrative examples, a graspingstrategy may call for a number of different robotic arms each havingdifferent end effectors or combinations of end effectors, or a graspingstrategy may involve activating a combination of end effectors availableon a single robotic arm. Any suitable end effector (or number orcombination of end effectors) may be utilized, including, but notlimited to, soft robotic effectors, vacuum effectors, electro-adhesioneffectors, and mechanical or electromechanical effectors. Soft roboticend effectors may generally include flexible structures that may bemanipulated between various orientations. The structures may includesilicon bodies or other flexible material. Manipulation of the flexiblematerial may be achieved through use of flexible actuators such as airmuscles (e.g., contractile or extensional devices operated bypressurized air movement relative to filling or emptying a pneumaticbladder), electro-active polymers (e.g., polymers which change size orshape when stimulated by an electric field), or ferrofluids (e.g.,fluids having suspended ferro-magnetic particles capable of altering asize or shape of the fluid volume when subjected to a magnetic field).Vacuum end effectors may grasp items using suction. Electro-adhesion endeffectors can include an array of electrodes arranged along a flexibleor rigid substrate capable of applying a charge (akin to staticelectricity) that can adhere an item to the substrate portions that arein contact with the item. Mechanical or electromechanical end effectorsmay include pinchers, claws, grippers, or other rigid components thatmay be actuated relative to one another for grasping an item. Other endeffectors may also be utilized to facilitate additional graspingtechniques. For example, a magnetic or electromagnetic end effector maybe useful for grasping items having ferro-magnetic materials.

A grasping strategy can also include an indication of a level ofintensity with which the robotic arm is to operate a specific endeffector. For example, for a mechanical or electromechanical pincher, agrasping strategy may include an amount of force (e.g., a fixed level ora varying profile) that the pincher is to exert during the graspingoperation. Intensity corollaries for other end effectors may includeamount of suction for vacuum end effectors, strength of magnetic fieldsfor magnetic or electromagnetic end effectors, current or charge exertedin an electro-adhesion end effector, or level of air pressure exerted toactuate a soft robotic end effector.

The human-based grasping strategy module 720 can provide a mechanism forobtaining information about a manner in which a human would grasp aparticular item. In some embodiments, the human-based grasping strategymodule 720 may facilitate observation of a human's grasping approach.For example, sensors (such as in the sensor package 16 of FIG. 1) can beconfigured to detect information while a human is grasping a targetitem. In an illustrative example, a human wearing a glove or othergarment with fiducial markers and/or sensors grasps a particular item.The motion of the human's approach is tracked based on the fiducialmarkers (e.g., as detected by optical sensors), and othercharacteristics are tracked based on the sensors (e.g., pressuresensors, tactile sensors, or other sensors may provide data regardingdetails about an amount and location of pressure that the human exertson the item throughout the process, shear forces or other indiciatranslatable into an amount that the object slips in the human's hand,or other information that may be useful in determining a graspingstrategy). A grasping strategy is developed by selecting an end effectortype, orientation, and intensity that is most capable of mimicking themotion and grip exhibited by the human. In some embodiments, real-timefeedback and control of a robotic arm can be provided for a humanperforming a grasping operation (e.g., haptic feedback or a visualdisplay showing the operation of a robotic arm in response to thehuman's grasping operation). In some embodiments, the human-basedgrasping strategy module 720 provides an interface for a human todirectly give instructions for how the robotic arm 12 should pick up orgrasp a particular item 40. For example, an illustrative user interfaceis described or shown in FIG. 8. In some embodiments, the human-basedgrasping strategy module 720 can provide a mechanism for a human to posethe robotic arm 12 in an orientation or position so as to instruct therobotic arm 12 how to grasp a target item. In some embodiments, thehuman-based grasping strategy module 720 can provide a virtualenvironment in which a human can perform or direct a grasping action foran item to facilitate machine learning of information for learning,developing, and/or determining a grasping strategy for the robotic arm12 to grasp a target item.

The constraints module 725 can handle information about factors that mayaffect or constrain a grasping strategy independent of the identity ofthe item to be grasped. As an illustrative example, the constraintsmodule 725 may access information about a receptacle into which agrasped item is to be placed so that end effectors that are too large tofit in the receptacle will be eliminated as options for the graspingstrategy. In general, the constraints module may provide informationabout a grasping or receiving environment for the grasped item, such astray types available for initial presentation for grasping, a size of areceptacle or a receiving zone, a location of a receptacle or receivingzone (such as on the ground, three feet off the ground, five feet fromthe grasping environment, etc.), a type of receiving environment (suchas a tray, shipping box, inventory holder, etc.), a designatedorientation of the grasped item when released in the receivingenvironment (such as if the grasped item is to be presented to a humanoperator in a particular ergonomically compliant orientation or if theitem is to be placed in a particular orientation to facilitatesubsequent automated actions).

In some embodiments, the constraints module 725 may provide informationabout whether a grasped item can be successfully released following aparticular initial approach to grasping the item. As an illustrativeexample, a coffee mug may have a closely-sized box that requires thehandle to be facing a side of the box rather than a top or bottom of thebox. The constraints module may thus use this information to eliminategrasping strategies (or provide information to the grasping strategyselection module 730) that would grip the coffee mug by the handle andresult in the mug being released in the box so that the handle is facingupward. Similarly, the constraints module may determine or indicate anamount of time the different grasping strategies would entail (which maybe useful information for the grasping strategy selection module 734 inselecting one grasping strategy over another). In another example, theconstraints module 725 may be updated by feedback from the graspingstrategy evaluation module 740. For example, the constraints module 725may provide information derived from other feedback, such as damagereports from third parties (e.g., a customer receiving an order, asupplier of an order, shipping personnel tasked with transporting anordered item), which may be useful in determining viability of graspingstrategies.

The grasping strategy selection module 730 can determine a graspingstrategy for a particular item. For example, the grasping strategyselection module 730 may utilize information from any or all of theattribute detection module 710, the database query module 715, thehuman-based grasping strategy module 720, the constraints module 725,and the grasping strategy evaluation module 740 to determine a graspingstrategy for a particular item and the environments in which the item isto be grasped, moved, and/or released. In addition to determining how anitem is to be grasped, or as an alternative, the grasping strategyselection module 730 may be involved in determining whether to graspsomething using a robotic arm 12. For example, if the attributedetection module 710 detects damage to an item 40, the grasping strategyselection module 730 may instruct an appropriate response, such asselecting a grasping strategy that includes refraining from grasping thedamaged item and locating another item of the same type that isundamaged instead.

Based on the grasping strategy selected by the grasping strategyselection module 730, the grasping strategy instruction module 735 mayprovide appropriate instructions to the robotic arm and other componentsto effectuate the grasping strategy. For example, the grasping strategyinstruction module 735 may instruct movement of a mobile drive unitcarrying an inventory holder to a station having a robotic arm, provideinstructions to cause a shipping container to be placed in a receivingzone for the robotic arm, and instruct the robotic arm to perform aseries of actions to carry out a grasping strategy that facilitatesmoving an inventory item from the inventory holder to the shippingcontainer.

The grasping strategy evaluation module 740 can evaluate the success ofthe grasping strategy instructed by the grasping strategy instructionmodule 735. For example, the grasping strategy evaluation module 740 mayevaluate different factors of success of a grasping operation, such as,but not limited to, whether the robotic arm was able to grasp andmaintain a grip of the target item, a time elapsed to complete thegrasping action, and an indication of whether the target item reachedthe intended destination and/or in the intended configuration. Suchfactors may be utilized, for example, for ranking a preference ofgrasping strategies for certain items and/or item characteristics. Thedatabase update module 745 can update records and/or grasping strategiesstored in databases, such as the item database 37 and/or the itemgrasping database 36 of FIG. 1. For example, the database update module745 may update an item database 37 to update a virtual model of an itemhaving a certain SKU based on sensed attributes determined by theattribute detection module 710 for another item of that SKU. In thisway, attributes of an item may be updated over time, by informationgathered by different combinations of sensors or different instances ofattribute detection. This may be useful, as attributes detected by asingle instance of sensing may be based on a single face of an item orotherwise yield less than all information available about that item. Thedatabase update module 745 may also update grasping strategies forparticular items (e.g., stored in the item grasping database 36 of FIG.1), such as in response to information received from the graspingstrategy evaluation module 740, the human-based grasping strategy module720, the constraints module 725, or any combination thereof. In thisway, a strategy that was successfully implemented by one robotic arm inone location may be rapidly deployed for implementation of a robotic armin another location in the same workspace or another inventory systemhaving access to database 36.

The database update module 745 may additionally or alternatively updateother databases, such as a vendor's item database that containsinformation about items the vendor has provided through the inventorysystem. In some embodiments, updating the vendor database may facilitateimprovements in a manner in which items are stored in inventory holdersas a result of a greater availability of information about the inventoryholders and the items to be stored. As an illustrative example, an itemthat is determined to be of a specific size based on detected attributeinformation may allow the item to be more efficiently stored in asmaller receptacle and/or along with other items, instead of allocatinga larger receptacle for the item based on an inaccurate assumption ofthe size of the inventory item. In another illustrative example,updating a vendor database may permit savings and shipping cost based onselecting a box more appropriately sized for the shipping item that isdetermined or verified, at least in part, relative to initial storeddata from sensors used for item characterization or attribute detection.

FIG. 8 illustrates an example user interface for obtaining human inputfor a grasping strategy. The user interface includes a screen showingvarious selection options that a human user can select to provide inputabout a grasping strategy for a particular item, e.g., a coffee cup 802.In a first row at 804, the user has options for different end effectors,such as a mechanical pincher 806, soft robotics 808, vacuum 810, orelectro-adhesion 812. The user can select one or more of the endeffectors as illustrated by the selection boxes 814. In a second row at820, the user can select an approach for picking up the desired item802. For example, the presented options may include an angled or lateralapproach 822, a top-down approach 824, an underneath approach 826, or amultiple arm approach 828 (in which multiple arms are used for graspingand moving the item 828). The user can select an approach (or more thanone approach in the case of multiple robotic arms) as illustrated by theselection box 818. At 830, a visualization of the selected graspingstrategy is presented in which the coffee cup is to be grasped at anangle in a lateral approach by a combination of a mechanical pincher andvacuum based on the selections made at 814 and 818.

FIG. 9 illustrates an example of a station having a robotic arm 912. Thestation 900 can represent a station 50 described elsewhere herein. Thestation 900 can include a picking area 902, a grasping area 904, arelease area 906, and a packing area 908. The mobile drive unit 910(e.g., mobile drive unit 20 described elsewhere herein) can bring aninventory holder 914 (e.g., inventory holder 30 described elsewhereherein), to the picking area 902. An operator 916 in the picking area902 can remove an inventory item 940 from the inventory holder 914 andtransfer the inventory item 940 to the grasping area 904 (e.g., to atray 918). The tray 918 may be one of a plurality of trays in thegrasping area 904. In some embodiments, the plurality of trays 918 aremoveable so as to bring empty trays into position to be filled by theoperator 916 and trays bearing items 40 into a position to be accessedby the robotic arm 912. Multiple and/or movable trays may allow therobotic arm 912 and the operator and 916 to work at different rates andmay reduce wait times in a workflow at the station 900. Multiple traysmay also facilitate simultaneous, independent operation of multiplerobotic arms 912 and/or picking operators 916. In some embodiments,different trays may be provided having differing characteristics thatfacilitate grasping different types of items (e.g., one tray type withsoft lining for grasping fruit and another tray type with features moresuited to grasping books).

The robotic arm 912 may receive instructions to perform a particulargrasping strategy to move an inventory item 940 from a tray 918 and intothe release area 906, e.g., into a box 922 positioned on a shelf 924A-Dof a stadium shelving unit 920. The shelves 924A-D of the stadiumshelving unit 920 may be different widths, which may permit the roboticarm 912 to access a variety of boxes from above in a manner that alsopermits the boxes to be accessed along a common vertical face in thepacking area 908. Conveyance mechanisms 928 may be provided on any ofthe shelves 924A-D of the stadium shelving unit 920 to move filled boxesfrom the release area 906 to the packing area 908, e.g., to within reachof a worker 926 in the packing area 908 for subsequent operations suchas packing a box 922 with completed orders for shipping. Although asingle conveyance mechanism 928 is depicted in FIG. 9 so as to avoidconfusion in the drawing, any number of conveyance mechanisms 928 of anytype (e.g., conveyor, pusher, puller) on any shelf 924A-D may beprovided to move boxes 922 to the packing area 908. Furthermore,although a stadium shelving unit 920 is depicted in FIG. 9, any otherform of transfer mechanism may be utilized to transfer a box or otherreceptacle with an item received from a robotic arm from a release area906 to a packing area 908. As illustrative examples, the stadiumshelving unit 920 may be augmented or replaced by a conveyor system(e.g., for moving empty and/or loaded boxes relative to the robotic arm912) or a set of inventory holders movable by mobile drive units.Transfer mechanisms between the release area 906 and the packing area908 may provide a degree of separation between human operators (e.g.,916 and 926) and the robotic arm 912, for example, to facilitatecompliance with safety measures.

In some embodiments, the station 900 can include a sensor package 16such as described in FIG. 1 with respect to sensor package 16. Forexample, the robotic arm 912—or other structure associated with thestation 900—may include such sensors for identifying items 940 (and/orcharacteristics thereof), boundaries of boxes 922 in the release area906 (such as to facilitate release by the robotic arm of the item 940 ina correct box 922), and/or for detecting motions and/or signalsperformed by an operator 916 (such as an alternate human input device 38of FIG. 1). Hence, as may be appreciated, grasping strategies may bedefined and/or refined at stations specifically designated fordetermining grasping strategies (e.g., FIG. 1) and/or at stationsdesignated for other inventory functions, such as picking or stowing,e.g., with respect to inventory holders 914 and/or mobile drive units910 (e.g., FIG. 9).

FIG. 10 illustrates a box 1022 (such as may correspond to a box 922 ofFIG. 9) having corner pieces 1030 having fiducial markers 1032. Thecorner pieces 1030 can retain flaps of the box open for loading by therobotic arm 912. For example, the corner pieces 1030 may be T- orL-shaped to retain perpendicular flaps in position relative to oneanother. Corner pieces may be constructed of any material. The cornerpieces 1030 can include any type of fiducial marker, including, but notlimited to barcodes, colors, patterns, alphanumeric text or codes,symbols, other indicia. Such fiducial markers 1032 may provideadditional points of reference for sensors used in conjunction with therobotic arm 912 to identify particular boxes in accordance withinstructions for moving the robotic arm 912.

FIG. 11 illustrates a cross-sectional view of a tray 1118, such as maycorrespond to a tray 918 of FIG. 9. The tray can include a layer orsurface of deformable material 1136. For example, the deformablematerial may be foam or gel or any other deformable material that candeform under pressure applied by a robotic arm during an approach thatincorporates an element passing underneath or behind the target item.The tray 1118 can additionally, or alternatively, include a fabric layerproviding a surface along which the target item can be moved. Forexample, such a layer of fabric 1136 may provide appropriate frictionconditions to facilitate the robotic arm gripping the target item from aposition on the tray or moving, e.g., sliding, the target item along thetray to a position where the robotic arm can get an appropriate grip onthe target item (e.g., sliding the target item from a lower horizontalposition up to an angled position so that a robotic arm can grip theitem from a different orientation more in line with a particulargrasping strategy). In some embodiments, the tray 1118 includes one ormore grooves 1138, which may provide an additional or alternative spacefor a component of the robotic arm to approach the target item at leastpartially from underneath. Other alternatives are also possible,including ribs or ridges or projections that would make the graspingsurface contoured in such a way as to facilitate grasping of an item atleast partially from underneath the item by the robotic arm 912.

FIG. 12 illustrates several components of an inventory system with whicha robotic arm 1212 can interact. The robotic arm 1212 can move inventoryitems 1240 to or from open boxes 1244, completed boxes 1246, trays 1218,drawers 1242, inventory holders 1214 (such as may store inventory itemsaccessible for fulfilling orders and/or store inventory items sortedinto compiled orders) or associated components of inventory holders 1214(such as bins, totes, slots, or compartments), delivery or othertransport vehicles 1250, conveyance mechanisms 1248 (such as spirals orother conveyors, chutes, or ramps), pallets 1254 (and/or any structure1256 supporting the pallet 1254 or receptacles 1252 stacked on thepallet), and/or any other any other elements of an inventory system. Therobotic arm 1212 may interact with human operators 1234 and/or otherrobotic arms 1221 to accomplish such movements of inventory items 1240and/or to facilitate other functions involving the human operators 1234or other robotic arms 1221. Furthermore, in addition to (or asalternatives to) grasping or otherwise facilitating movement ofindividual inventory items 1240, the robotic arm 1212 may also grasp orotherwise facilitate movement of other components of the inventorysystem, including, but not limited to the tray 1218, open boxes 1244,closed boxes 1246, drawers 1242, bins, totes, pallets 1254, structures1256, or receptacles 1252 shown in FIG. 12.

As one of many illustrative examples, the robotic arm 1212 can interactwith a tray 1218, such as moving an inventory item 1240 to or from thetray 1218, e.g., to an open box 1244 or from a human operator 1234. Insome aspects, a robotic arm 1212 may remove or replace a drawer 1242 orother bin, tote, or receptacle with respect to an inventory holder 1214(such as may be moved by mobile drive unit 1210). The robotic arm 1212may move the drawer 1242 to or from an ergonomically appropriateposition at which a human operator 1234 (or other robotic arm 1221) mayperform prior or subsequent actions with the item 1240. In some aspects,the robotic arm 1212 may directly move items 1240 to or from storagelocations within the inventory holder 1214. The robotic arm 1212 maygrasp and move open boxes 1244 (such as empty or partially-filled boxes)or closed and sealed boxes 1246 to or from pallets 1254, inventoryholders 1214, conveyance mechanisms 1248, and/or delivery vehicles 1250.

FIG. 13 illustrates aspects of an example environment 1300 forimplementing aspects in accordance with various embodiments. As will beappreciated, although a Web-based environment is used for purposes ofexplanation, different environments may be used, as appropriate, toimplement various embodiments. The environment includes an electronicclient device 1302, which can include any appropriate device operable tosend and receive requests, messages, or information over an appropriatenetwork 1304 and convey information back to a user of the device.Examples of such client devices include personal computers, cell phones,handheld messaging devices, laptop computers, set-top boxes, personaldata assistants, electronic book readers, and the like. The network caninclude any appropriate network, including an intranet, the Internet, acellular network, a local area network or any other such network orcombination thereof. Components used for such a system can depend atleast in part upon the type of network and/or environment selected.Protocols and components for communicating via such a network are wellknown and will not be discussed herein in detail. Communication over thenetwork can be enabled by wired or wireless connections and combinationsthereof. In this example, the network includes the Internet, as theenvironment includes a Web server 1306 for receiving requests andserving content in response thereto, although for other networks analternative device serving a similar purpose could be used as would beapparent to one of ordinary skill in the art.

The illustrative environment includes at least one application server1308 and a data store 1310. It should be understood that there can beseveral application servers, layers, or other elements, processes orcomponents, which may be chained or otherwise configured, which caninteract to perform tasks such as obtaining data from an appropriatedata store. As used herein the term “data store” refers to any device orcombination of devices capable of storing, accessing, and retrievingdata, which may include any combination and number of data servers,databases, data storage devices and data storage media, in any standard,distributed or clustered environment. The application server can includeany appropriate hardware and software for integrating with the datastore as needed to execute aspects of one or more applications for theclient device, handling a majority of the data access and business logicfor an application. The application server provides access controlservices in cooperation with the data store and is able to generatecontent such as text, graphics, audio and/or video to be transferred tothe user, which may be served to the user by the Web server in the formof HyperText Markup Language (“HTML”), Extensible Markup Language(“XML”) or another appropriate structured language in this example. Thehandling of all requests and responses, as well as the delivery ofcontent between the client device 1302 and the application server 1308,can be handled by the Web server. It should be understood that the Weband application servers are not required and are merely examplecomponents, as structured code discussed herein can be executed on anyappropriate device or host machine as discussed elsewhere herein.

The data store 1310 can include several separate data tables, databasesor other data storage mechanisms and media for storing data relating toa particular aspect. For example, the data store illustrated includesmechanisms for storing information which can be used by modulesdescribed herein, such as resource scheduling information 1312, routeplanning information 1314, segment reservation information 1316, and/orinventory information 1318. Furthermore, the management module 15 ofFIG. 2 and/or FIG. 7, and all modules included therein, can be stored indata store 1310. It should be understood that there can be many otheraspects that may need to be stored in the data store, such as for pageimage information and to access right information, which can be storedin any of the above listed mechanisms as appropriate or in additionalmechanisms in the data store 1310. The data store 1310 is operable,through logic associated therewith, to receive instructions from theapplication server 1308 and obtain, update or otherwise process data inresponse thereto.

Each server typically will include an operating system that providesexecutable program instructions for the general administration andoperation of that server and typically will include a computer-readablestorage medium (e.g., a hard disk, random access memory, read onlymemory, etc.) storing instructions that, when executed by a processor ofthe server, allow the server to perform its intended functions. Suitableimplementations for the operating system and general functionality ofthe servers are known or commercially available and are readilyimplemented by persons having ordinary skill in the art, particularly inlight of the disclosure herein.

The environment in one embodiment is a distributed computing environmentutilizing several computer systems and components that areinterconnected via communication links, using one or more computernetworks or direct connections. However, it will be appreciated by thoseof ordinary skill in the art that such a system could operate equallywell in a system having fewer or a greater number of components than areillustrated in FIG. 13. Thus, the depiction of the system 1300 in FIG.13 should be taken as being illustrative in nature and not limiting tothe scope of the disclosure.

The various embodiments further can be implemented in a wide variety ofoperating environments, which in some cases can include one or more usercomputers, computing devices or processing devices which can be used tooperate any of a number of applications. User or client devices caninclude any of a number of general purpose personal computers, such asdesktop or laptop computers running a standard operating system, as wellas cellular, wireless and handheld devices running mobile software andcapable of supporting a number of networking and messaging protocols.Such a system also can include a number of workstations running any of avariety of commercially-available operating systems and other knownapplications for purposes such as development and database management.These devices also can include other electronic devices, such as dummyterminals, thin-clients, gaming systems and other devices capable ofcommunicating via a network.

Most embodiments utilize at least one network that would be familiar tothose skilled in the art for supporting communications using any of avariety of commercially-available protocols, such as TransmissionControl Protocol/Internet Protocol (“TCP/IP”), Open SystemInterconnection (“OSI”), File Transfer Protocol (“FTP”), Universal Plugand Play (“UpnP”), Network File System (“NFS”), Common Internet FileSystem (“CIFS”) and AppleTalk. The network can be, for example, a localarea network, a wide-area network, a virtual private network, theInternet, an intranet, an extranet, a public switched telephone network,an infrared network, a wireless network, and/or any combination thereof.

In embodiments utilizing a Web server, the Web server can run any of avariety of server or mid-tier applications, including Hypertext TransferProtocol (“HTTP”) servers, FTP servers, Common Gateway Interface (“CGI”)servers, data servers, Java servers and business application servers.The server(s) also may be capable of executing programs or scripts inresponse requests from user devices, such as by executing one or moreWeb applications that may be implemented as one or more scripts orprograms written in any programming language, such as Java®, C, C# orC++, or any scripting language, such as Perl, Python or TCL, as well ascombinations thereof. The server(s) may also include database servers,including without limitation those commercially available from Oracle®,Microsoft®, Sybase® and IBM®.

The environment can include a variety of data stores and other memoryand storage media as discussed above. These can reside in a variety oflocations, such as on a storage medium local to (and/or resident in) oneor more of the computers or remote from any or all of the computersacross the network. In a particular set of embodiments, the informationmay reside in a storage-area network (“SAN”) familiar to those skilledin the art. Similarly, any necessary files for performing the functionsattributed to the computers, servers or other network devices may bestored locally and/or remotely, as appropriate. Where a system includescomputerized devices, each such device can include hardware elementsthat may be electrically coupled via a bus, the elements including, forexample, at least one central processing unit (“CPU”), at least oneinput device (e.g., a mouse, keyboard, controller, touch screen orkeypad) and at least one output device (e.g., a display device, printeror speaker). Such a system may also include one or more storage devices,such as disk drives, optical storage devices and solid-state storagedevices such as random access memory (“RAM”) or read-only memory(“ROM”), as well as removable media devices, memory cards, flash cards,etc.

Such devices also can include a computer-readable storage media reader,a communications device (e.g., a modem, a network card (wireless orwired), an infrared communication device, etc.) and working memory asdescribed above. The computer-readable storage media reader can beconnected with, or configured to receive, a computer-readable storagemedium, representing remote, local, fixed, and/or removable storagedevices as well as storage media for temporarily and/or more permanentlycontaining, storing, transmitting, and retrieving computer-readableinformation. The system and various devices also typically will includea number of software applications, modules, services or other elementslocated within at least one working memory device, including anoperating system and application programs, such as a client applicationor Web browser. It should be appreciated that alternate embodiments mayhave numerous variations from that described above. For example,customized hardware might also be used and/or particular elements mightbe implemented in hardware, software (including portable software, suchas applets) or both. Further, connection to other computing devices suchas network input/output devices may be employed.

Storage media and computer readable media for containing code, orportions of code, can include any appropriate media known or used in theart, including storage media and communication media, such as but notlimited to volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage and/or transmissionof information such as computer readable instructions, data structures,program modules or other data, including RAM, ROM, Electrically ErasableProgrammable Read-Only Memory (“EEPROM”), flash memory or other memorytechnology, Compact Disc Read-Only Memory (“CD-ROM”), digital versatiledisk (DVD) or other optical storage, magnetic cassettes, magnetic tape,magnetic disk storage or other magnetic storage devices or any othermedium which can be used to store the desired information and which canbe accessed by the a system device. Based at least in part on thedisclosure and teachings provided herein, a person of ordinary skill inthe art will appreciate other ways and/or methods to implement thevarious embodiments.

The specification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the disclosure asset forth in the claims.

Other variations are within the spirit of the present disclosure. Thus,while the disclosed techniques are susceptible to various modificationsand alternative constructions, certain illustrated embodiments thereofare shown in the drawings and have been described above in detail. Itshould be understood, however, that there is no intention to limit theinvention to the specific form or forms disclosed, but on the contrary,the intention is to cover all modifications, alternative constructionsand equivalents falling within the spirit and scope of the invention, asdefined in the appended claims.

The use of the terms “a” and “an” and “the” and similar referents in thecontext of describing the disclosed embodiments (especially in thecontext of the following claims) are to be construed to cover both thesingular and the plural, unless otherwise indicated herein or clearlycontradicted by context. The terms “comprising,” “having,” “including,”and “containing” are to be construed as open-ended terms (i.e., meaning“including, but not limited to,”) unless otherwise noted. The term“connected” is to be construed as partly or wholly contained within,attached to, or joined together, even if there is something intervening.Recitation of ranges of values herein are merely intended to serve as ashorthand method of referring individually to each separate valuefalling within the range, unless otherwise indicated herein and eachseparate value is incorporated into the specification as if it wereindividually recited herein. All methods described herein can beperformed in any suitable order unless otherwise indicated herein orotherwise clearly contradicted by context. The use of any and allexamples, or exemplary language (e.g., “such as”) provided herein, isintended merely to better illuminate embodiments of the invention anddoes not pose a limitation on the scope of the invention unlessotherwise claimed. No language in the specification should be construedas indicating any non-claimed element as essential to the practice ofthe invention.

Preferred embodiments of this disclosure are described herein, includingthe best mode known to the inventors for carrying out the invention.Variations of those preferred embodiments may become apparent to thoseof ordinary skill in the art upon reading the foregoing description. Theinventors expect skilled artisans to employ such variations asappropriate and the inventors intend for the invention to be practicedotherwise than as specifically described herein. Accordingly, thisinvention includes all modifications and equivalents of the subjectmatter recited in the claims appended hereto as permitted by applicablelaw. Moreover, any combination of the above-described elements in allpossible variations thereof is encompassed by the invention unlessotherwise indicated herein or otherwise clearly contradicted by context.

All references, including publications, patent applications and patents,cited herein are hereby incorporated by reference to the same extent asif each reference were individually and specifically indicated to beincorporated by reference and were set forth in its entirety herein.

What is claimed is:
 1. A non-transitory computer-readable storage mediumhaving stored therein instructions that, when executed by one or moreprocessors of a computer system, cause the computer system to performoperations comprising at least: for each inventory item of a pluralityof inventory items: determining an identity of the inventory item to begrasped from among identities previously stored for inventory items;accessing, based on the determined identity of the inventory item, arecord from an item database including characteristics of the inventoryitem; accessing, based at least in part on the characteristics of theinventory item, grasping strategies from an item grasping database; andresponsive to the inventory item not having a grasping strategyassociated therewith: generating a grasping strategy for the inventoryitem based at least in part on at least one of (i) characteristics ofthe inventory item that are at least one of stored in the record ordetected by one or more sensors; (ii) input from a human operatorreceived through a human-based grasping strategy module for grasping theinventory item; (iii) grasping strategies for items with characteristicssimilar to the characteristics of the inventory item; (iv) physicalconstraints associated with a location in which the inventory item is tobe grasped or placed; or (v) information about a success of graspingstrategies that have been previously employed; instructing a roboticmanipulator to perform the generated grasping strategy for the inventoryitem; and storing the generated grasping strategy in association withthe characteristics of the inventory item in the item grasping database.2. The non-transitory computer-readable storage medium of claim 1,wherein the operations further comprise at least: receiving evaluationinformation about a success of the generated grasping strategy; andupdating the item grasping database based on the evaluation information.3. The non-transitory computer-readable storage medium of claim 2,wherein updating the item grasping database based on the evaluationinformation comprises, responsive to the evaluation informationindicating that the generated grasping strategy was successful,designating the generated grasping strategy stored in association withthe characteristics of the inventory item as the designated graspingstrategy for the inventory item.
 4. The non-transitory computer-readablestorage medium of claim 2, wherein updating the item grasping databasebased on the evaluation information comprises, responsive to theevaluation information indicating that the generated grasping strategywas unsuccessful, identifying the generated grasping strategy stored inassociation with the characteristics of the inventory item as anineffective grasping strategy.
 5. The non-transitory computer-readablestorage medium of claim 1, wherein generating a grasping strategy forthe inventory item comprises generating a grasping strategy that isdifferent from grasping strategies that are associated with thecharacteristics of the inventory item in the item grasping database asineffective grasping strategies.
 6. A method comprising, under thecontrol of one or more computer systems configured with executableinstructions: determining an identity of an inventory item to be graspedfrom a grasping environment; accessing, based on the determined identityof the inventory item, a record in an item database about the inventoryitem, the record including information about one or more geometriccharacteristics of the inventory item; receiving, from a sensor, one ormore detected attributes of the inventory item in the graspingenvironment; determining, based at least in part on the one or moregeometric characteristics and the one or more detected attributes, anorientation of the inventory item in the grasping environment;generating a grasping strategy using a robotic manipulator for theinventory item based at least in part on the determined orientation ofthe inventory item in the grasping environment, wherein generating thegrasping strategy includes selecting an end effector from amongdifferent types of available end effectors; and storing the generatedgrasping strategy in association with at least one of the one or moregeometric characteristics, the one or more detected attributes, or thedetermined orientation of the inventory item in an item graspingdatabase.
 7. The method of claim 6, wherein generating a graspingstrategy for the inventory item based at least in part on the determinedorientation of the inventory item in the grasping environment comprises:instructing an alignment of the end effector based on the determinedorientation of inventory item.
 8. The method of claim 6, wherein the endeffector comprises at least one of a mechanical end effector, anelectromechanical end effector, a vacuum end effector, anelectro-adhesion end effector, a magnetic end effector, or a softrobotic end effector.
 9. The method of claim 6, wherein selecting an endeffector to use in the grasping strategy comprises selecting at leastone of (i) a type of end effector, (ii) an intensity at which the endeffector is to be operated, or (iii) an orientation at which the roboticmanipulator is to approach the inventory item; and wherein the selectionis based at least in part on at least one of: (i) one or more of the oneor more geometric characteristics of the inventory item in the record(ii), one or more of the one or more detected attributes of theinventory item, or (iii) the determined orientation of inventory item.10. The method of claim 6, wherein generating a grasping strategy forthe inventory item comprises generating the grasping strategy based atleast in part on physical constraints associated with the graspingenvironment.
 11. The method of claim 6, wherein generating a graspingstrategy for the inventory item comprises generating the graspingstrategy based at least in part on information about a success ofgrasping strategies that have been previously employed.
 12. The methodof claim 6, wherein generating a grasping strategy for the inventoryitem comprises generating the grasping strategy based at least in parton grasping strategies for items with at least one of: (i) geometriccharacteristics similar to the one or more geometric characteristics ofthe inventory item, (ii) attributes similar to the one or more detectedattributes of the inventory item, or (iii) orientations similar to thedetermined orientation of the inventory item.
 13. The method of claim 6,wherein generating a grasping strategy for the inventory item comprisesgenerating the grasping strategy based at least in part on observationinformation from one or more sensors observing a human perform agrasping operation.
 14. The method of claim 6, wherein generating agrasping strategy for the inventory item comprises generating thegrasping strategy based on at least one of instructions received via auser interface or actions observed in a virtual environment.
 15. Amethod comprising, under the control of one or more computer systemsconfigured with executable instructions: receiving attribute informationabout an inventory item, the attribute information including at leastone of an identity of the inventory item or information about physicalcharacteristics of the inventory item; receiving observation informationfrom one or more sensors observing a hand or other appendage of a humanperform a grasping operation for the inventory item; generating, basedat least in part on the observation information, a grasping strategy forgrasping the inventory item using a robotic manipulator; and storing thegenerated grasping strategy in a record in an item grasping database inassociation with the attribute information.
 16. The method of claim 15,further comprising: instructing the robotic manipulator to execute thegenerated grasping strategy to grasp a target inventory item in responseto receiving an indication that the target inventory item comprisesattribute information in common with the attribute information of theinventory item.
 17. The method of claim 15, wherein the observationinformation includes information about an orientation at which the handor other appendage of the human performs the grasping operation for theinventory item.
 18. The method of claim 15, wherein the observationinformation includes information about a pressure, shear force, or otherphysical characteristic with which the hand or other appendage of thehuman performs the grasping operation for the inventory item.
 19. Themethod of claim 15, wherein receiving observation information from oneor more sensors observing a hand or other appendage of the human performa grasping operation for the inventory item comprises receiving theobservation information from the one or more sensors on a glove worn bythe human.
 20. The method of claim 15, further comprising providingreal-time feedback and control of the robotic manipulator as the humanperforms the grasping operation.